Statistical Mechanics of On-line Learning
نویسندگان
چکیده
We introduce and discuss the application of statistical physics concepts in the context of on-line machine learning processes. The consideration of typical properties of very large systems allows to perfom averages over the randomness contained in the sequence of training data. It yields an exact mathematical description of the training dynamics in model scenarios. We present the basic concepts and results of the approach in terms of several examples, including the learning of linear separable rules, the training of multilayer neural networks, and Learning Vector Quantization.
منابع مشابه
Optimisation of on–line principal component analysis
Various techniques, used to optimise on-line principal component analysis, are investigated by methods of statistical mechanics. These include local and global optimisation of node-dependent learning-rates which are shown to be very efficient in speeding up the learning process. They are investigated further for gaining insight into the learning rates’ timedependence, which is then employed for...
متن کاملLETTER TO THE EDITOR Globally optimal on-line learning rules for multi-layer neural networks
We present a method for determining the globally optimal on-line learning rule for a soft committee machine under a statistical mechanics framework. This rule maximizes the total reduction in generalization error over the whole learning process. A simple example demonstrates that the locally optimal rule, which maximizes the rate of decrease in generalization error, may perform poorly in compar...
متن کاملOptimization of the Asymptotic Property of Mutual Learning Involving an Integration Mechanism of Ensemble Learning
Abstruct– We propose an optimization method of mutual learning which converges into the identical state of optimum ensemble learning within the framework of on-line learning, and have analyzed its asymptotic property through the statistical mechanics method.The proposed model consists of two learning steps: two students independently learn from a teacher, and then the students learn from each o...
متن کاملEffect of Slow Switching in On-line Learning for Ensemble Teachers
We have analyzed the generalization performance of a student which slowly switches ensemble teachers. By calculating the generalization error analytically using statistical mechanics in the framework of on-line learning, we show that the dynamical behaviors of generalization error have the periodicity that is synchronized with the switching period and the behaviors differ with the number of ens...
متن کاملStatistical Mechanics of Mutual Learning with a Latent Teacher
We propose a mutual learning with a latent teacher within the framework of on-line learning, and have analyzed its dynamical behavior through the statistical mechanics method. The proposed model consists of two learning steps: two students independently learn from a teacher, and then the students learn from each other through the mutual learning. A teacher is not used in the mutual learning, so...
متن کاملGlobally Optimal On-line Learning Rules
We present a method for determining the globally optimal on-line learning rule for a soft committee machine under a statistical mechanics framework. This work complements previous results on locally optimal rules, where only the rate of change in generalization error was considered. We maximize the total reduction in generalization error over the whole learning process and show how the resultin...
متن کامل